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1  Summary 

The  purpose  of  this  program  is  to  address  the  development  of  algorithms  for  adaptive 
processing  of  multi-sensor  data,  employing  feedback  to  optimize  the  linkage  between 
observed  data  and  sensor  control.  The  envisioned  multi-modal  adaptive  system  is 
applicable  for  intelligence,  surveillance,  and  reconnaissance  (ISR)  in  general 
environments,  addressing  base  and  port  security,  as  well  as  urban  and  suburban  sensing 
during  wartime  and  peace-keeping  operations.  Of  significant  importance  for  current  and 
anticipated  DoD  activities,  the  ISR  system  is  designed  to  detect  asymmetric  threats,  with 
the  goal  of  recognizing  unusual  behavior  or  activities.  Technologies  and  systems 
developed  under  this  effort  will  be  designed  for  semi-automated  scene  awareness,  with 
the  objective  of  recognizing  behavior  that  appears  atypical  (e.g.  atypical  object  motion, 
and  dynamic  characteristics  of  people  and  vehicles).  Leveraging  our  previously 
developed  technology,  SIG  is  developing  second-generation  methods  to  adaptively  learn 
the  statistics  of  dynamic  object  behavior  in  video,  while  focusing  on  defining  system 
requirements  for  sensor  deployment  by  using  field  data  (vs.  highly  controlled  indoor 
data).  SIG  is  also  working  closely  with  its  subcontractor,  Lockheed  Martin,  to  integrate 
additional  technologies,  such  as  object  classification  and  recognition,  to  provide  a  more 
robust  and  discriminative  system.  Additionally,  SIG  is  cooperating  with  the  Navy’s 
China  Lake  facility  to  collect  representative  data  for  a  deployed  system,  and  to  specify 
requirements  and  features  necessary  of  such  a  system.  Finally,  SIG  is  coordinating  with 
Integrian  on  prototype  development  schedules  and  product  integration  requirements,  and 
defining  a  joint  marketing  and  commercialization  strategy  for  such  products. 

SIG  is  aggressively  pursing  follow-on  funding  and  technology  transition  opportunities  for 
persistent  surveillance  applications.  In  particular,  under  related  IRAD  efforts,  we  are 
working  to  address  current  shortfalls  and  develop  new  methods  for  activity  recognition  of 
vehicles  and  dismounts  in  persistent  airborne  EO/IR  video  imagery.  An  efficient 
framework  is  under  development  for  joint  classification  and  target  tracking  (JCT).  The 
joint  posterior  density  over  target  poses  (e.g.  position,  velocity,  heading)  and  target  type 
is  recursively  estimated  via  a  Bayesian  formulation.  It  is  assumed  that  appearance 
measurements  provide  information  indicative  of  target  class  and  kinematic  measurements 
for  indirect  measure  of  target  pose.  An  importance  sampling  approach  is  used  to 
efficiently  incorporate  both  types  of  measurements  in  refining  estimation  of  the  joint 
distribution.  The  methods  are  presented  in  the  context  of  providing  real-time  analytic 
metadata  to  reduce  data  transmission  requirements  in  support  of  persistent  video-based 
surveillance.  Included  in  this  metadata  are  the  joint-posterior  distributions  for  target 
tracks  and  class  which  are  utilized  by  an  active  learning  cueing  management  framework 
to  optimally  task  the  appropriate  sensor  modality  to  cued  regions  of  interest.  Moreover, 
this  active  learning  approach  also  facilitates  analyst  cueing  to  help  resolve  track 
ambiguities  in  complex  scenes.  We  intend  to  leverage  SIG’s  active  learning  with 
analyst  cueing  under  future  efforts  with  ONR  and  other  DoD  agencies.  Obtaining  long¬ 
term  accurate  target  tracks  is  a  key  requirement  for  activity  modeling.  We  propose 
adapting  methods  for  activity  manifold  modeling,  such  that  the  posterior  distributions  of 
vehicle  tracks  themselves  may  be  ultimately  transformed  into  space-time  probability 
manifolds.  Such  representations  will  enable  further  application  of  the  active  learning 
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framework  for  semi-automated  activity  recognition  with  limited  analyst  cueing  for 
anomaly  and  threat  notification. 

2  Recent  Technical  Developments 

During  this  reporting  period,  we  have  completed  the  integration  of  our  robust  detection  & 
tracking  engine  with  the  asymmetric  threat  behavior  analysis  capabilities  that  we  have 
developed  under  this  C2CS  effort.  We  have  implemented  a  software  application  that  can 
process  real-time  or  archived  color  video  data  to  extract  objects  tracks  and  features  and  to 
identify  in  real-time  any  objects  that  exhibit  anomalous  behavior  characteristics.  The  SIG 
team  attended  the  PI  Gathering  at  ONR  in  May  2008  and  presented  our  current  results  as 
well  as  providing  a  demonstration  of  the  integrated  software  behavior  detection 
application.  In  recent  months,  we  have  also  made  significant  progress  in  extending  our 
initial  behavior  analysis  work  to  include  long-term  track  information  in  addition  to  the 
instantaneous  object  positions  and  trajectories  used  in  our  initial  work. 

2.1  Full  integration  of  tracking  engine  and  anomalous  behavior  Detection 

Over  the  course  of  the  last  few  months,  SIG  has  focused  its  attention  on  the  integration  of 
or  behavior  analysis  algorithms  with  our  real-time  object  detection  and  tracking  engine  to 
create  a  Color  video  ISR  system  that  detects  unusual  behavior  or  activities.  This  system 
provides  real-time  operation  and  supports  detection  in  cluttered  environments.  At  the 
same  time,  it  can  help  to  deal  with  situations  where  there  is  limited  bandwidth  to  analysts 
through  the  use  of  intelligent  compression. 

The  core  engine  of  this  application  is  the  flexible  multi-hypothesis  tracking.  A  screen 
capture  is  shown  of  this  application  in  Figure  1.  This  application  supports  operation  with 
either  live  or  archived  color  video  data  and  allows  detection  and  tracking  of  objects  as 
small  as  4  pixels  in  the  target  domain.  The  system  can  track  through  total  occlusions  and 
can  reacquire  after  as  much  as  two  seconds  of  complete  occlusion.  This  systems 
demonstrates  repeatable  tracking  accuracy  and  is  robust  to  occlusions,  lighting,  and 
inclement  weather.  It  provides  98.7%  data  association  accuracy  with  occlusions  under 
testing  and  was  demonstrated  on  a  0.052/object-minute  divergence  rate  where  tracked 
objects  would  diverge  during  occlusion. 

This  system  is  an  efficient  real-time  implementation  and  operates  at  15  fps  on  real-time 
streaming  color  video.  The  system  operates  even  faster  on  archived  video  -  when  not 
limited  by  the  real-time  camera  constraint  and  can  operate  in  excess  of  20  fps  on  a  single 
processor.  The  system  also  enables  intelligent  video  compression  as  it  identifies  all 
objects  of  interest  in  the  foreground.  When  tracking  distributions  are  employed  to 
allocate  data  bits,  the  system  provides  as  much  as  39x  raw  data  compression  ratio  in  the 
target  domain  using  simple  compression.  Performance  as  high  as  llOx  encoded  data 
compression  ratios  possible  using  more  sophisticated  compression  approaches. 
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This  application  was  demonstrated  at  the  PI  gathering  that  was  held  in  Arlington  on  May 
12-14,2008. 
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Figure  1:  Screen  shot  of  the  integrated  tracking  and  anomalous  behavior  detection  application. 

2.2  Full  track  behavior  analysis 

Our  initial  behavior  modeling  was  based  on  instantaneous  position  and  velocity  only. 
This  approach  characterized  behavior  in  terms  of  position  and  velocity  and  then 
compared  this  to  past  behaviors.  The  approached  used  the  joint  probability  for  improved 
discrimination.  Because  most  unusual  actions  are  suspicious,  this  approach  was  able  to 
capture  desired  anomalous  activities  such  as  trespassing,  speeding,  loitering,  and  flow 
control  violations.  This  approach  provided  a  generalized  solution  that  avoided  the  pitfalls 
of  rule-based  systems  since  the  behavior  distributions  were  learned  from  the  scene 
directly.  During  testing,  this  approach  demonstrated  correct  detection  of  all  anomalous 
activity  in  scripted  video  test  sequences  within  several  frames  and  also  detected 
unscripted  objects  with  anomalous  behavior. 

During  the  recent  performance  period,  our  modeling  efforts  are  being  extended  to  full 
motion  trajectories.  This  new  approach  allows  us  to  characterize  behavior  in  terms  of 
entire  track  history  provided  by  object  tracking  engine.  For  each  object,  the  trajectory  is 
parameterized  by  arc-length  to  provide  invariance  to  time  sampling.  This  approach  is  a 
learning  based  strategy  where  we  focus  on  conditional  probabilities  of  behavior  P(Y  IX), 
rather  than  full  joint  P(X,Y)  (where  Y  are  the  behaviors  and  X  are  the  observes  object 
states).  We  use  boosting  and  logistic  regression  (specifically,  the  LogitBoost  algorithm) 
for  training  and  object  behavior  classification.  The  approach  was  tested  on  outdoor  video 
surveillance  sequences  collected  at  China  Lake  facility  and  results  are  shown  in  Figure  2. 
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The  initial  results  were  based  on  an  approach  that  used  only  object  trajectory  - 
subsequent  improvements  will  incorporate  appearance  of  objects  as  well  -  and  our  goal  is 
>95%  classification  accuracy.  Figure  3  also  shows  a  typical  video  frame  from  the  testing 
data  that  shows  a  typical  object  with  an  indication  of  different  potential  trajectories. 
Additional  work  will  also  incorporate  active  learning  and  additional  behaviors. 


Vehicle  Vehicle  Person  Person 
parked  exits  entets  exits 
facility  facility 


Figure  2:  testing  results  from  the  full-trajectory  behavior  analysis  on  the  China  Lake  data  set. 


Figure  3:  Typical  video  from  form  the  China  Lake  data  set  with  indications  of  conditional  object 

trajectories. 
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2.3  Technology  transition  to  Persistent  Surveillance  Applications 

We  have  begun  transition  of  the  technology  developed  under  this  C2CS  effort  to  multiple 
other  funded  research  efforts.  The  ONR-funded  SIG  C2CS  effort  has  transitioned  into 
funded  programs  in  the  applications  of  Army  Persistent  Surveillance  (Constant  Hawk 
future  IR  capability  -  JIEDDO  funded),  Air  Force  persistent  surveillance  (funded  by 
AFRL),  and  airborne  IED  detection  (funded  by  NVESD).  The  technology  transitions  total 
$450k  per  year  non-ONR  transition  funding.  For  the  persistent  surveillance  applications, 
the  technology  will  be  used  to  support  real-time  (or  forensic)  image  and  video  analysis 
using  the  C2CS-developed  modeling  and  Bayesian  tracking  technology.  In  this 
application,  analysts  will  provide  feedback  to  algorithms  to  resolve  tracking  and 
classification  ambiguities.  An  integrated  system  will  cue  analysts  to  regions  of  ambiguity 
in  algorithm  inferences,  and  any  cued  regions  will  be  prioritized  according  to  relative 
information  gain.  In  addition,  there  is  the  potential  for  transition  of  the  intelligent  video 
compression  technology  that  allocates  full  resolution  to  critical  scene  data  and 
compresses  background  imagery  for  context.  The  Army  effort  will  apply  this  C2CS 
technology  (tracking,  behavior  analysis  and  active  learning)  to  Army  UAV  data  sets  and 
the  Air  Force  effort  will  apply  this  C2CS  technology  to  the  CFIF  data  set  (funded 
through  AFRF). 
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Figure  4:  Typical  image  from  the  IED  target  data  based  along  with  resulting  plots  showing 
improvements  in  both  analyst  accuracy  (fewer  false  alarms  during  analysis)  as  well  as  improved 
efficiency  (4-6  time  s  faster  analysis). 

The  Army  NVESD  airborne  IED  detection  application  will  use  active  learning  and  video 
processing  to  real-time  streaming  video  imagery  for  target  detection.  The  work  will 
involve  the  development  &  testing  on  borne  IR  data  sets  funded  through  NVESD.  Initial 
results  have  demonstrated  the  effectiveness  of  active  learning  for  improving  analyst 
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performance  and  efficiency.  A  typical  image  is  shown  in  Figure  4  where  a  region  of 
interest  has  been  identified  for  analyst  labeling,  and  specific  results  are  shown  for  this 
data  set  indicating  improvements  in  analyst  accuracy  and  efficiency.  We  will  report  the 
specific  details  of  these  technology  transitions  separately  using  the  Transition  report  form 
that  has  been  provided  by  ONR  to  SIG. 

3  Future  Directions 

During  the  next  reporting  period,  we  will  continue  work  on  using  the  concepts  of 
compressive  sensing  to  address  some  of  the  fundamental  challenges  regarding  the 
detection  of  asymmetric  threats.  In  particular,  these  concepts  can  be  used  to  optimize  a 
system  to  extract  the  most  possible  information  about  possible  targets  and  their  behavior 
under  the  constraints  of  system  processing  and  data  bandwidth  constraints.  We  will 
continue  the  work  on  transition  of  the  C2CS  technology  for  airborne  persistent 
surveillance  applications,  as  well  as  for  integration  with  systems  that  use  analyst-in-the- 
loop  technology  for  analyst  cueing  to  provide  more  robust  tracking  and  behavior  analysis 
in  scenarios  where  difficult  conditions  such  as  poor  resolution,  environment  or  light 
conditions  lead  to  tracking  or  data  association  ambiguities.  In  addition  to  this  near-term 
work,  we  are  continuing  to  work  toward  transitioning  the  research  to  other  funded 
research  efforts  in  addition  to  those  identified  above. 
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